Routine Administration | Oracle Application Server 10g: J2EE Deployment and Administration

The administration of 10g AS takes skills of both a web administrator and an Oracle DBA. Of course, the more complex the installation type and architecture, the more complex and time-consuming the tasks of the administrator. Fortunately, once the system is installed and configured, most tasks can easily be managed via the web-based Application Server Control utility or by operating system shell scripts and commands.

Administrative Duties

The duties of a 10g AS administrator largely depend on the applications being supported. Oracle Portal, PLSQL Web Toolkit, and Forms and Reports applications will be more Oracle database centric. J2EE applications will require more Java and OC4J skills. Although some duties will be the same no matter what kind applications are supported, those administrators supporting J2EE applications will be primarily focused on the following duties:

Installing and configuring software on development, test, and production environments. Each environment should be built in a consistent manner and controlled changes should progress from development to testing and finally, to production.
Applying necessary patches for both functionality and security vulnerabilities. While it may not be critical to apply a patch for a feature you're not using, you should apply security patches as soon as they're available. Oracle Security Alerts can be found at http://otn.oracle.com/deploy/security/alerts.htm .
Ensuring that web servers and the software are up and running correctly. Scripts to restart software after machine reboot should be placed in /etc/rc3.d directories and tested prior to an application going live. Work with the system administrator to implement these scripts.
Deploying J2EE applications for testing and production environments. In most cases, developers should have the ability to deploy their application code to the development environment. However, testing and production environments should be under control of a Release Management authority to control which code is deployed. Oftentimes the web administrator is responsible for deploying code to testing and production environments. Deploying J2EE applications is covered in Chapter 18.
Managing log files is a necessary function. If a log file gets too large and fills up a file system, the software can hang. Additionally, writing to very large log files is a performance hit. 10g AS has introduced several new control utilities that keep these files under control. The Apache community also has several mechanisms for managing and analyzing OHS logs.
Backup and recovery cannot be stressed enough. For installations using a metadata repository, DBA skills are needed to back up the database. These are typically performed by the DBA and are outside the scope of this book. However, for J2EE applications, filesystem backups and Distributed Configuration Management (DCM) backups should be enough. Be sure to test the backup procedures regularly.

Caution

Many corruptions of OC4J containers occur during deployments. If you're using a standalone OC4J container, take a TAR backup of the j2ee directory and the subdirectories below it before deploying to a production environment. Oftentimes it's faster to restore or even reinstall a standalone OC4J container than to fix it. For an embedded OC4J container such as 10g AS, take a DCM backup.

Other duties of the J2EE administrator typically center on troubleshooting, performance tuning, and security. These activities are difficult at times, but can be made easier by Oracle's graphical-administration utility, Application Server Control.

Application Server Control Utility

Oracle recommends the web-based ASC utility for most administrative tasks. Originally referred to as an Enterprise Manager utility in 9.0.2, this product wasn't related to the OEM tools DBAs often use so it has been renamed in 9.0.4.

Starting and Accessing the Application Server Control Utility

To start the ASC utility, simply issue the emctl start iasconsole command, as follows :

 $ emctl start iasconsole TZ set to US/East-Indiana Oracle Enterprise Manager 10g Application Server Control 9.0.4.0.0 Copyright (c) 2002, 2003 Oracle Corporation. All rights reserved. Starting Oracle 10g Application Server Control ..... .. started. $

Once the process is started, you may access it at http://hostname.domain:1810 or http://mike.wessler. name :1810 for this example.

You'll be prompted for the ias_admin password that you entered during the installation, as seen in Figure 16-18.

Figure 16-18: ias_admin login prompt

After the ias_admin password is authenticated, the next page that appears is the Application Server Farm home page. This screen shows every instance associated with the infrastructure and its status (up or down). If any component of an instance is down, the entire instance will be listed as down, although components in it may actually be running. At this point there's only one instance running ( 904mt1 ), as you can see in Figure 16-19.

Figure 16-19: Application Server Farm home page

After clicking an instance listed on the Farm home page, the Instance home page appears. This page shows the following options:

Instance name
Operating-system metrics for memory and CPU utilization
Each component of the instance such as OC4J home instance or OHS
Options to stop, start, or restart the entire instance or an individual component

These options are shown in Figure 16-20.

Figure 16-20: Instance home page

In Chapters 17 and 18 we'll cover OC4J setup and administration in greater detail. For now, these options can be found by clicking the OC4J home page.

Oracle HTTP Server Management with the Application Server Control Utility

OHS settings are particularly important, both for security and performance. The home page for the HTTP server is shown in Figure 16-21.

Figure 16-21: Oracle HTTP_Server home page

This page allows you to stop, start, or restart the OHS server. However, the links to performance metrics, error logs, virtual hosts , and administration are the most useful.

The Error Logs page is useful because it allows you to select which logs to view and can combine the results into one listing. This is important because rather than manually shifting through multiple log files in different directories to trace a problem, you can have the relevant information in one consolidated screen. Though you still can manually view logs (sometimes that's useful), this utility is a good time-saver. Logs that you can view are Business Components for Java (BC4J), DCM, Enterprise Manager (ASC), HTTP Server, LogLoader, Oracle Process Management Notification (OPMN), port tunneling, OUI, and the home OC4J application.

The Virtual Hosts page identifies each IP and server name and port combination that OHS is listening on if it's SSL enabled. It's good to make sure OHS is listening to only the IPs and ports that you've set up; shutdown addresses you're not using. This screen also allows the creation of additional virtual hosts with Create and Create Like buttons as well as deletions.

The Administration page provides links to Server Properties, MIME settings, PL/SQL ( mod_plsql Web Toolkit), and Advanced Server Properties pages. The Server Properties page is the most useful page for changing OHS settings, as you can see in Figure 16-22.

Figure 16-22: HTTP Server Properties page

This page lists the location for the Server Root directory, which is the basis for all other Apache files and directories. It also allows you to specify parameter values that are translated into Apache directives. For example, the ServerAdmin directive is the point-of-contact email address listed on error screens; this is listed on the ASC screen as "Administrator E-Mail." Other items that can be configured are

Document Root. This is the default location for static content such as HTML files.
Listening Addresses and Ports. What ports and IPs the OHS will listen on. This is the same information as listed on the Virtual Hosts page.
Logging. Log file locations, names , and logging levels (for example, informational, warningthe default, critical, emergency). It also has links to both error and access logs that take you back to the Error Log page.
Client request and connection handling. These are settings for the individual httpd processes. They map to the following Apache directives, as shown in Table 16-1.

Table 16-1: Apache Directive Mappings
Server Control Title	Apache Directive and Value	Apache Documentation Description
Maximum Requests Processed Simultaneously	MaxClients 150	The MaxClients directive sets the limit on the number of simultaneous requests that can be supported; not more than this number of child server processes will be created.
Request Timeout (seconds)	TimeOut 300	TimeOut: The number of seconds Apache will wait to receive a request or a response before Apache will send a time out.
Limit Requests Handled by Each Child Server Process	MaxRequestsPerChild 0	MaxRequestsPerChild: The number of requests each child process is allowed to process before the child dies.
Allow Multiple Requests per Connection	KeepAlive On	KeepAlive: Whether or not to allow persistent connections (more than one request per connection). Set to "Off" to deactivate.
Connection Timeout (seconds)	KeepAliveTimeout 15	KeepAliveTimeout: Number of seconds to wait for the next request from the same client on the same connection.
Limit Requests Connection	MaxKeepAliveRequests 100	MaxKeepAliveRequests: The maximum per number of requests to allow during a persistent connection. Set to to allow an unlimited amount.

As you can see in the preceding table, the ASC tool essentially takes configuration options and maps them to the appropriate Apache directive. Excellent Apache documentation can be found at http://httpd.apache.org/docs/ and also within configuration files as comments.

Though a detailed account of OHS and Apache configuration settings are outside the scope of this book, you'll find that most Apache settings are managed in the httpd.conf file located in $ORACLE_HOME/Apache/Apache/conf . To access this and other configuration files within the ASC tool, navigate to the Advanced Server Properties page under the HTTP_Server administration page. The files and their locations, which can be manually configured, are shown in Figure 16-23.

Figure 16-23: Advanced Server Properties files

Clicking any of these files will open the file in the ASC editor. You may edit the file and then click Save or Revert. After making a change to this (or any configuration file), and clicking Save, DCM will propagate the changes back to the infrastructure repository. This ensures that the repository and the physical files are in sync. You'll also be prompted to restart the relevant component (OHS or OC4J) for the changes to take effect.

Additional Configuration and Stopping the Application Server Control Utility

New in the 10g AS Instance home page are links to Process Management, J2EE Applications, Ports, and Infrastructure. The most powerful is Process Management, which allows you to edit the opmn.xml file. This file controls the OPMN processes. Within the file you can change ports, logging levels, security policies, and component status. Make a backup of it before editing it because this is a key file. The Ports page displays which ports are in use and has links for configuring them. The Infrastructure page has important configuration links for Identity Management, central OEM management, and cluster management. Since you're doing a standalone J2EE installation, the Infrastructure page isn't highly important.

To stop the ASC, use the emctl stop iasconsole command, as follows:

 $ emctl stop iasconsole TZ set to US/East-Indiana Oracle Enterprise Manager 10g Application Server Control 9.0.4.0.0 Copyright (c) 2002, 2003 Oracle Corporation. All rights reserved. Stopping Oracle 10g Application Server Control ... ... Stopped. $

It's generally recommended that you should have only have the ASC utility and web pages running when you need them. Performance and security are the primary reasons for keeping it shut down except for when changes need to be made. First, unnecessarily having the background processes so that you can keep the iasconsole running takes memory that could be used for other processes. Second, by default the web page runs in HTTP rather than the more secure HTTPS mode in a well-documented nonstandard port (1810). This represents yet another avenue for hackers to explore so you're better off shutting it down except for when it's needed. We cover making it more secure by SSL in Chapter 20.

Command-Line Utilities

Every action executed by the ASC is translated into an equivalent series of commands executed by OPMN and DCM. The ASC is really just a fast way to generate command strings for the 10g Application Server. Just as the Oracle database tool Enterprise Manager generates SQL commands for the database, so does the ASC. Therefore, anything you can do with the ASC you can do with OPMN and DCM.

Oracle Process Management Notification

OPMN system is a background process that manages all the individual components for a given 10g AS instance. When an installation is set up, specific components are configured and expect to be started. For example, a J2EE installation would have at a minimum OHS and the single home OC4J instance. As J2EE applications are deployed, more OC4J instances are created (for example Payroll and HR). Each of these interrelated processes is managed by OPMN. If a process dies, OPMN will attempt to restart it. It continually polls the processes it expects to be up and restarts them if they die or become unreachable. Think of OPMN as the managing processes for component processes within an individual 10g AS instance.

Though OPMN is a background process, it has a command-line interface that can be used to stop or start all the processes for a given installation. The opmnctl utility is a handy way to start or stop the software without having to use the web GUI ASC.

Startup, Status, and Shutdown Commands

The opmnctl utility can be used to start every component within 10g AS except for the Oracle metadata repository database and the web-based ASC utility. To start all the processes of an individual installation, issue the opmnctl startall command, as follows:

 $ opmnctl startall opmnctl: starting opmn and all managed processes... $

This will start every component in the installation such as OHS and all the OC4J processes. This may take a minute or two to run because there are multiple actions occurring behind the scenes (allocating Java Virtual Machines, starting processes, and so on). When scripted, this command is often followed by a sleep command that allows it to complete.

To stop all the processes of an individual installation, issue the opmnctl stopall command, as shown here:

 $ opmnctl stopall opmnctl: stopping opmn and all managed processes... $

Since OPMN monitors all the processes for its instance, it makes sense that it can provide a status for each component. To see what components are running and their process ID (PID), enter opmnctl status . To get more verbose status information including memory utilization, uptime, and ports used, issue the opmnctl status l command, as shown here:

 $ opmnctl status Processes in Instance: 904mt1.mike.wessler.name -------------------+--------------------+-------+--------- ias-component       process-type          pid  status -------------------+--------------------+-------+--------- OC4J                home                30930  Alive HTTP_Server         HTTP_Server         31207  Alive dcm-daemon          dcm-daemon          30611  Alive LogLoader           logloaderd            N/A  Down $

As you can see in the preceding code sample, OPMN breaks down the status by component type. Now that you know what specific components are configured, you can stop, start, or restart a component individually (similar to the OHS server).

 $ opmnctl stopproc ias-component=HTTP_Server opmnctl: stopping opmn managed processes... $ opmnctl startproc ias-component=HTTP_Server opmnctl: starting opmn managed processes... $ opmnctl restartproc ias-component=HTTP_Server opmnctl: restarting opmn managed processes...

In cases for which there's more than one ias-component of a given type, such as multiple OC4J instances, you need to also identify the process-type . The following command would restart the home OC4J instance, but would leave any other OC4J instances running:

 $ opmnctl restartproc ias-component=OC4J process-type=home opmnctl: restarting opmn managed processes... $

The opmnctl command is located in the $ORACLE_HOME/opmn/bin directory. You may find it a time-saver to add this directory to the PATH variable so that you don't need to type the full path location. This can be added to the .profile or done via the command line, as shown here:

 $ export PATH=$PATH:$ORACLE_HOME/opmn/bin

Logs for the OPMN process are located in $ORACLE_HOME/opmn/logs , as follows:

 $ ls $ORACLE_HOME/opmn/logs dcm-daemon~dcm-daemon~dcm~1  ipm.log                     ons.log   states HTTP_Server~1                OC4J~home~default_island~1  opmn.log $

Startup and Shutdown Scripts

You may find it useful to write start and stop scripts using the OPMN commands so that any trusted administrator can start, stop, or restart 10g AS if you're not available and don't have the time to fully train them. These scripts can also be used on machine reboots to start the software. Like any script or procedure, be sure to test them thoroughly before trusting your production system to them.

The start script is as follows:

 $ more start_10gas.ksh #!/usr/bin/ksh # Name: start_10gas.ksh # By: M. Wessler 1/25/2004 # Script to start 10g AS J2EE installation # Assumes no infrastructure associated with this instance export DISPLAY=mike.wessler.name:1 export ORACLE_HOME=/u01/app/oracle/product/9.0.4mt1/OraHome1 export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/opmn/bin:$ORACLE_HOME/dcm/bin:. echo "Starting all processes via OPMN" $ORACLE_HOME/opmn/bin/opmnctl startall echo "Checking process status" $ORACLE_HOME/dcm/bin/dcmctl getstate echo "\nShould see all processes Up and Sync status is True" $

Notice that you don't start the web-based ASC utility; this product isn't needed for when you run the software normally so you can leave it alone.

Running this startup script is simple.

 $ start_10gas.ksh Starting all processes via OPMN opmnctl: starting opmn and all managed processes... Checking process status Current State for Instance:904mt1.mike.wessler.name     Component               Type          Up Status     In Sync Status ====================================================================== 1   home                    OC4J          Up            True 2   HTTP_Server             HTTP_Server   Up            True Should see all processes Up and Sync status is True $

The equivalent stop script is as follows:

 $ more stop_10gas.ksh #!/usr/bin/ksh # Name: stop_10gas.ksh # By: M. Wessler 1/25/2004 # Script to stop 10g AS J2EE installation # Assumes no infrastructure associated with this instance export DISPLAY=mike.wessler.name:1 export ORACLE_HOME=/u01/app/oracle/product/9.0.4mt1/OraHome1 export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/opmn/bin:$ORACLE_HOME/dcm/bin:. echo "Stopping all processes via OPMN" $ORACLE_HOME/opmn/bin/opmnctl stopall sleep 5 echo "\nShould not see any running application server processes\n" ps -ef  grep -i oracle  grep -v ksh  grep -v grep  grep -v ps $

Running the stop script is simple.

 $ stop_10gas.ksh Stopping all processes via OPMN opmnctl: stopping opmn and all managed processes... Should not see any running application server processes root      1527  1509  0 Jan28 pts/0    00:00:00 su - oracle $

There other command options with opmnctl . To get a full listing of options, issue opmnctl help and opmnctl usage .

Distributed Configuration Management

DCM works in conjunction with OPMN and has the primary responsibility of synchronization of the instance components (for example, OC4J and OHS) and with the repository. Though OPMN works internally within the instance, DCM communicates instance updates externally to other instances and the infrastructure. This is especially important when the mid- tier instance is part of a farm (attached to an infrastructure) or part of a cluster. DCM makes sure that the configuration files on the file system are in sync with the repository and any updates are propagated accordingly . Even if the installation isn't associated with an infrastructure instance to become part of a farm, updates to files are stored in a file-based repository just for that standalone instance.

When initially setting up an instance, you may prefer to use the ASC because it's faster and easier for complex tasks. When you want to "bounce" (stop or start) the entire application server instance, you may want to use the opmnctl stop and opmnctl startall commands. However, we find DCM's command-line utility useful for the following tasks:

Checking the status of the application server instance
Synchronizing configuration files with the repository
Making backups of the instance configuration
Stopping and starting individual components within the instance

Status and Synchronization

The dcmctl utility is located in $ORACLE_HOME/dcm/bin and should be added to your path along with the opmnctl utility.

To check the status of the application server instance, issue the dcmctl getstate command. Starting in release 9.0.4, the results from dcmctl commands are given with verbose output by default so it's no longer necessary to specify the v flag, as shown here:

 $ dcmctl getstate Current State for Instance:904mt1.mike.wessler.name     Component               Type          Up Status     In Sync Status ========================================================================= 1   home                    OC4J          Up            True 2   HTTP_Server             HTTP_Server   Up            True $

DCM responds with a listing of each component name ( home and HTTP_Server ), the type of component it is ( OC4J or HTTP_Server ), if it's up or down, and if it believes the configuration files of the instance are in sync with the contents of the repository (either file-based or the Oracle infrastructure metadata database). This provides a way to identify what components are configured and if they're running. Though you could attempt to grep for http processes to check the HTTP_Server , it isn't possible to search for individual OC4J applications so this utility is very handy.

In the event that a component isn't in sync with the repository, DCM will use what is in the repository to update the files on the operating system. Given a choice, DCM "believes" what is in its repository rather than what is in the actual configuration files. To force a resync of the files with what is in the repository, issue the dcmctl resyncinstance command. If you want to overwrite the contents of the repository with the configuration files as they exist on the file system, issue dcmctl updateConfig .

Be careful not to use updateConfig when other administrators are using the ASC or dcmctl utilities. This will avoid consistency errors that can occur when multiple administrators are making simultaneous changes. Given that DCM handles updates automatically and that you should not be updating files manually outside the ASC editor, you should not have to issue these update and resync commands often.

Backups

DCM is also capable of taking backups of a repository configuration as an archive snapshot. This is useful in the event that you want to restore a configuration to a given point in time. Archive backups can also be exported and imported into different instances to essentially clone an instance. Here the repository is the local file-system repository located in $ORACLE_HOME/dcm/repository .

The administrator can configure DCM to take these archive snapshots automatically or on an ad hoc basis. If set to automatic, DCM will make an archive backup at the given interval, and prior to any changes being made. Auto archiving is turned on by default, but to verify that it's on issue the dcmctl set command, as shown here:

 $ dcmctl set Verbose:  true Sort:  false Warning:  true Debug:  true Default Timeout:  120 Auto Archive Count :  15 $

You see that Auto Archive Count is greater than 0, therefore you know the auto archiving feature is on. In this case, DCM will keep up to 15 automatic backups before overwriting them. To make a manual archive backup, issue the createArchive command, as shown here:

 $ dcmctl createArchive -arch 904mt1bk-02012004_1

The archive isn't physically stored on the file system as a single file by default like a database export .dmp file or a tar file backup. Archives are stored in the $ORACLE_HOME/dcm/repository directory in the archive file. To create a physical archive file of a specific backup, you must export the archive using the exportarchive command. Here you make a backup of your 904mt1bk-02012004_1 archive, as follows:

 $ dcmctl exportarchive -arch 904mt1bk-02012004_1 f /home/oracle/archives/904mt1bk-02012004_1 $ ls -altr /home/oracle/archives/904mt1bk-02012004_1 -rw-r-----    1 oracle   dba        370770 Feb  1 11:53  /home/oracle/archives/904mt1bk-02012004_1 $

To determine what archive snapshots are stored and available for retrieval, issue dcmctl listarchives . This will list all archives in the local repository archive.

To apply a backup archive, you can either import from an exported physical archive file (created earlier) with the importarchive command or with the applyarchiveto command for an archive stored in the local repository, as shown here:

 $ dcmctl importarchive -arch 904mt1restore-02012004_1 -f /home/oracle/archives/ 904mt1bk-02012004_1 $ dcmctl applyarchiveto -arch 904mt1bk-02012004_1

In the first example, you restored from the exported archive file and named the configuration 904mt1restore-02012004_1 . In the second example, you restored with the backup located in the local repository.

Though there's no substitute for good operating-system level and database backups, these DCM backups are important. Be sure to make a backup before making configuration changes and keep auto archiving turned on.

Starting and Stopping Components

Individual components can be stopped or started using dcmctl . This is often the case if the file was updated and you want the change to take effect (you should use the ASC for updates). It's also the case if a process is hung and it needs bounced, or if you want only some processes running (for example, you want to shut down OC4J demos).

To stop or start a component, it's necessary to identify either the component name (with co flag) in the command or component type (with ct flag). Generally, the co component name flag is most useful when operating on individual OC4J component instances where you don't want to impact all the running OC4J components. If no name or component is specified, all the components for the instance are impacted although OPMN processes aren't effected.

For example, to stop and start the OHS component, the following commands could be entered:

 $ dcmctl stop -co HTTP_Server Current State for Instance:904mt1.mike.wessler.name     Component               Type          Up Status     In Sync Status ============================================================================ 1   HTTP_Server             HTTP_Server   Down          True $ dcmctl start -co HTTP_Server Current State for Instance:904mt1.mike.wessler.name     Component               Type          Up Status      In Sync Status ============================================================================ 1   HTTP_Server             HTTP_Server   Up             True $

There is also a restart option if you only want to bounce a component, as shown here:

 $ dcmctl restart -co HTTP_Server Current State for Instance:904mt1.mike.wessler.name     Component               Type          Up Status     In Sync Status ============================================================================ 1   HTTP_Server             HTTP_Server   Up            True $

Starting in 9.0.4, Oracle recommends using opmnctl to start or stop individual components of an instance rather than dcmctl . The purpose of DCM is to work in distributed environments rather than on the standalone instance. Therefore, if you're working in a standalone configuration, use opmnctl to manage individual processes. If you're working in part of a cluster, use dcmctl .

Logs for DCM are located in the $ORACLE_HOME/dcm/logs directory, as shown here:

 $ ls $ORACLE_HOME/dcm/logs busrule_logs  daemon_logs  dcmctl_logs  emd_logs $

There is more that can be done with the dcmctl utility, but much of it can be done more easily with the ASC. To get more information on the dcmctl utilities, issue dcmctl help .