| Trace files represent the best way to get detailed information about what Net8 is doing behind the scenes, and they are extremely helpful in troubleshooting situations. A trace file is a text file in which a Net8 facility records very detailed information about every event that occurs and every action taken. Trace files can grow to enormous sizes, and there's a fair bit of overhead involved in writing to them, so in normal operation, trace files aren't generated. Instead, you have to enable tracing when you need it to troubleshoot a problem, and then disable it afterwards. Several Net8 facilities can generate trace files. These are listed in Table 10.7 along with the parameters used to control tracing, and the configuration files in which you set those parameters. To enable tracing of a listener, for example, you set various parameters in the listener.ora file. Table 10.7. Net8 Trace File Parameters | Facility | Configuration File | Parameters | | N et8 Client and Server | sqlnet.ora | TRACE_DIRECTORY_CLIENT TRACE_DIRECTORY_SERVER TRACE_FILE_CLIENT TRACE_FILE_SERVER TRACE_LEVEL_CLIENT TRACE_LEVEL_SERVER TRACE_UNIQUE_CLIENT | | Listener | listener.ora | TRACE_DIRECTORY_ listener_name TRACE_FILE_ listener_name TRACE_FILELEN_ listener_name TRACE_FILENO_ listener_name TRACE_LEVEL_ listener_name TRACE_TIMESTAMP_ listener_name | | Oracle Names | names.ora | NAMES.TRACE_DIRECTORY NAMES.TRACE_FILE NAMES.TRACE_FUNC NAMES.TRACE_LEVEL NAMES.TRACE_UNIQUE | | Connection Manager | cman.ora | TRACING TRACE_DIRECTORY TRACE_FILELEN TRACE_FILENO TRACE_TIMESTAMP | | Names Control utility | sqlnet.ora | NAMESCTL.TRACE_LEVEL NAMESCTL.TRACE_FILE NAMESCTL.TRACE_DIRECTORY NAMESCTL.TRACE_UNIQUE | | TNS Ping | sqlnet.ora | TNSPING.TRACE_DIRECTORY TNSPING.TRACE_LEVEL | The manner in which tracing is invoked and controlled is relatively consistent across the various Net8 facilities, so in this chapter we are going to focus on just one: the Net8 client. We'll demonstrate an obscure connectivity problem that one of us recently encountered , and we'll show how trace files were used to resolve it. 10.3.1 A Hypothetical yet Real Problem The problem that we are going to demonstrate involves a failure to connect to a remote database. The net service name definition that we are using looks like this: DONNA_T.GENNICK.ORG = (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = donna.gennick.org)(PORT = 1523)) ) (CONNECT_DATA = (SERVICE_NAME = donna.gennick.org) ) ) The remote database runs on a Linux node named donna.gennick.org . The database is an Oracle8 i release, and it's configured for multi-threaded server (MTS). Here's what happens when we try to connect: C:\>sqlplus system/manager@donna_t SQL*Plus: Release 8.1.6.0.0 - Production on Thu Aug 24 21:15:05 2000 (c) Copyright 1999 Oracle Corporation. All rights reserved. ERROR: ORA-12545: Connect failed because target host or object does not exist The target host doesn't existthat seems like a straightforward enough message. Yet we know the host does exist because we can ping it: C:\> ping donna.gennick.org Pinging donna.gennick.org [10.11.12.16] with 32 bytes of data: Reply from 10.11.12.16: bytes=32 time<10ms TTL=255 Reply from 10.11.12.16: bytes=32 time<10ms TTL=255 Reply from 10.11.12.16: bytes=32 time<10ms TTL=255 Reply from 10.11.12.16: bytes=32 time<10ms TTL=255 Not only does the host exist, but the listener is up, running, and responding to tnsping s: C:\> tnsping donna_t TNS Ping Utility for 32-bit Windows: Version 8.1.6.0.0 - Production on 24-AUG-2000 21:16:50 (c) Copyright 1997 Oracle Corporation. All rights reserved. Attempting to contact (ADDRESS=(PROTOCOL=TCP)(HOST=donna.gennick.org)(PORT=1523)) OK (30 msec) A look at the listener log file on the remote host yields the following entry corresponding to our failed connection attempt. Notice that the status returned was 0, indicating that the connect attempt was successful: 24-AUG-2000 21:14:46 * (CONNECT_DATA=(SERVICE_NAME=donna.gennick.org)(CID= (PROGRAM=E:\Oracle\Ora81\bin\SQLPLUS.EXE)(HOST=JONATHAN)(USER=jonathan))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.11.12.13)(PORT=3740)) * establish * donna.gennick.org * 0 We can ping the remote host. We can ping the remote listener. Yet when we try to connect, we receive a message saying that the target host does not exist. On top of all that, the listener's log file indicates that the connection was successful. What gives? To find out, we can enable tracing for the Net8 client. 10.3.2 Trace File Locations and Names To enable tracing for the Net8 client, we need to set some parameters in the sqlnet.ora file. The default client trace filename is sqlnet.trc ( cli.trc on some systems), and the default location is the $ORACLE_HOME /network /trace directory. To change those values, we can place entries such as the following in the sqlnet.ora file: TRACE_FILE_CLIENT = host_problem.trc TRACE_DIRECTORY_CLIENT = e:\oracle\ora81\network\trace The sqlnet.ora file also supports TRACE_FILE_SERVER and TRACE_DIRECTORY_SERVER parameters. Those apply when you're tracing Net8's server functionality. For most client connectivity problems, you'll want to set the two client parameters shown here.  | Never make your trace directory the root directory ( C:\ for example) on a Windows system because, as a result, Net8 will not write the trace file that you are asking for. | | Setting the trace file location and name does not enable tracing. For tracing to occur, you must enable it by choosing a trace level. 10.3.3 Trace Levels A trace level is a setting that controls the amount of detail that gets written to a trace file. The default trace level for all Net8 facilities is OFF, which results in no trace data being generated. The other levels, in order of increasing levels of detail, are USER, ADMIN, and SUPPORT. There's no real semantic meaning to these keywords, at least not so far as we've ever been able to determine. Just realize that USER gets you some trace output, ADMIN gets you more, and SUPPORT will overwhelm you. When we enabled SUPPORT level tracing for the simple SQL*Plus connection attempt shown earlier, our resulting trace file was 820 lines long! That's a lot to read through. For this example, we'll use the ADMIN level, and we'll enable tracing at that level by placing the following line in our sqlnet.ora file: TRACE_LEVEL_CLIENT = ADMIN Now, with tracing enabled, all that's left to do is to generate the error by repeating our connection attempt. Then we can look in the trace file where we will (we hope) find some clue as to what is going wrong. 10.3.4 Trace File Contents Even at the ADMIN trace level, the trace file generated from our one connection attempt is 569 lines long. You can see why tracing is something that you don't want to leave enabled on a regular basis. If it takes 569 lines just to record the connection, imagine what it will take to record an entire session. One key to reading Net8 trace files is not to get too bogged down in the details. In our experience, it's never necessary to understand the full meaning of each and every line in the file. One approach to reading trace files is to scan them from top to bottom. Another approach is to locate the records in the trace file that correspond to the problem you encountered, and then work backward from those. In doing so, you're likely to spot an entry that gives you a clue as to what went wrong. In our example, we are trying to connect using a net service name of donna_t.gennick.org (the default domain is gennick.org ). The following lines in the trace file record the beginning of Net8's attempt to resolve that net service name. You can see here that Net8 is determining the name resolution methods to use: nnfgsrsp: Obtaining path parameter from names.directory_path or native_names. directory_path nnfgsrdp: entry nnfgsrdp: Setting path: nnfgsrdp: checking element ONAMES nnfgsrdp: checking element TNSNAMES nnfgsrdp: Path set nnfgrne: Going though read path adapters nnfgrne: Switching to ONAMES adapter nnfgrne: Original name: donna_t Following these are a large number of lines pertaining to name resolution attempts involving two Oracle Names servers. Net8 eventually switches to the second name resolution method, and the name is successfully resolved using the tnsnames.ora file: nnfgrne: Switching to TNSNAMES adapter nnfgrne: Original name: donna_t nnftqnm: entry nnfcagmd: entry nnfcagmd: Attribute name a.smd is a predefined meta type, syntax is 4. nnfcagmd: exit nncpldf_load_addrfile: initial load of names file E:\Oracle\Ora81\network\admin \tnsnames.ora nncpldf_load_addrfile: success nnftqnm: Using tnsnames.ora address (DESCRIPTION = (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = donna.gennick.org)(PORT = 1523))) (CONNECT_DATA = (SERVICE_NAME = donna.gennick.org))) for name donna_t.gennick.org nnfcraa: entry nnfgrne: Name successfully queried So far, so good. The protocol address and connect data shown in the trace file match what we have in our tnsnames.ora file. Everything looks good. Further on, we see these two lines: nttbnd2addr: port resolved to 1523 nttbnd2addr: looking up IP addr for host: donna.gennick.org This host address is correct, and in the trace file we can see that Net8 goes on to make the connection. Eventually, many lines later, we come to the following sequence: nscall: connecting... nttbnd2addr: entry nttbnd2addr: port resolved to 1027 nttbnd2addr: looking up IP addr for host: donna nttbnd2addr: *** hostname lookup failure! *** Here's our failure, and here you can see that the hostname has somehow changed from donna.gennick.org to donna . Without the proper domain, the name donna cannot be resolved. Notice the port number of 1027. That's not the listener's port number, and it may give you a clue as to what's going on here. Looking further back in the trace file, we find this unpretentious line nestled in among many others: nscall: redirected A redirect! This is where knowledge of Net8 architecture comes into play. The more familiar you are with how Net8 works internally, the better prepared you are to make sense of Net8 trace files. In our example, MTS is being used on the server. When MTS is used, connections are initially made to a listener and are then handed off (or redirected) to a dispatcher process. The listener accomplishes the redirect by passing a new protocol address back to the client. Part of this protocol address is a hostname. Where does the listener get the hostname? It gets it from the operating system. Sure enough, on this particular Linux system we find the following incorrect entry in the /etc/sysconfig /network : HOSTNAME=donna The system administrators (us in this case) haven't set the hostname correctly. The correct setting should be HOSTNAME=donna.gennick.org. The hostname in the net service name definition was correct, which explains why we could ping and tnsping the server. However, when the redirect was done, the listener picked up the hostname from the server and passed it back to the client. Unfortunately, that hostname was not correct and, at that point in the connection process, the error occurred. This was a very subtle problem, and you'd need a reasonable amount of networking, operating system, and Oracle experience to pick up on all the clues and solve it. However, this is the type of problem that you are likely to face if you are in deep enough to need tracing in the first place. Don't worry if a solution doesn't leap out at you on your first try. The more problems you solve, and the more you use Net8's tracing features, the more you'll learn about Net8. We always get a kick out of solving a knotty problem through a careful examination of Net8 trace files. Even if a trace file is no help to you, it may be just the thing that an Oracle support analyst needs to help you work through a problem. When you finish using the trace feature, be sure to disable it. You can do that by deleting the trace level parameter from the configuration file, or by setting the trace level to OFF. If you don't disable tracing, performance will suffer because Net8 needs to write all that information to disk, and eventually you'll run out of disk space. 10.3.5 Tracing and Other Net8 Features The example in this chapter shows how Net8's tracing feature can be used to diagnose client connectivity problems. Tracing can also be used with other Net8 features. If you're experiencing problems with Oracle Names, you can enable tracing for your Names server. Ditto with Connection Manager. While each facility has its own configuration file, the parameters used to control tracing are similar for all Net8 facilities. There are, however, a few trace features that are not universal. These include: -
The ability to generate unique trace filenames by appending a process ID to the filename -
The ability to control whether or not a timestamp is included with each trace record -
The ability to specify a trace file length and to rotate through a numbered set of trace files Oracle Names allows you to specify that trace filenames include a process ID by using the NAMES.TRACE_UNIQUE parameter. This results in a new trace file being created each time Names is restarted, which can be convenient if you prefer to work with many separate files instead of one huge one. The Names Control utility implements the same feature, so choosing which to use is really a matter of personal preference. If you are restarting Names a lot while working through a problem, using unique trace filenames reduces the amount of detail that you need to read through each time since you can always start with the most recent trace file. With Connection Manager and the Net8 Listener, you have the ability to control whether or not timestamps are written to trace files. The TRACE_TIMESTAMP parameter controls this feature. Our recommendation is to always include timestamp information. Timestamps can sometimes be helpful, and we can't think of any great gain to be had from omitting that sort of information. Connection Manager and the Net8 Listener also allow you to place a limit on trace file size using the TRACE_FILELEN parameter. If you do that, you also need to specify a value for the number of trace files that you want to create by using the TRACE_FILENO parameter. Connection Manager or the Net8 listener will then write trace information in a round- robin fashion to the specified number of trace files. Once the last file is written, Net8 starts over by overwriting the first file. As a result, you can leave tracing on for an extended period of time without fear of overrunning your disk. The number and size of the trace files control the amount of trace information on hand at any given time. Older trace information is aged out as newer trace information is written. | 
