| Sometimes Unix seems to be made up of a grab bag of applications and tools. There are tools to troubleshoot tools. And of course, there are several ways to accomplish the same task. When trying to solve a problem related to Samba, a good plan of attack is to use the following: Let's go over each of these one-by-one in the following sections. 12.1.1 Samba Logs Your first line of attack should always be to check the log files. The Samba log files can help diagnose the vast majority of the problems faced by beginning- to intermediate-level Samba administrators. Samba is quite flexible when it comes to logging. You can set up the server to log as little or as much information as you want. Using substitution variables in the Samba configuration file allows you to isolate individual logs for each system, share, or combination thereof. Logs are placed in /usr/local/samba/var/smbd.log and /usr/local/samba/var/nmbd.log by default. You can specify a log directory to use with the -l flag on the command line when starting the Samba daemons. For example: # smbd -l /var/log/samba # nmbd -l /var/log/samba Alternatively, you can override the location and name using the log file configuration option in smb.conf . This option accepts all the substitution variables, so you could easily have the server keep a separate log for each connecting client system by specifying the following: [global] log file = %m.log Another useful trick is to have the server keep a log for each service (share) that is offered , especially if you suspect a particular share is causing trouble. To do this, use the %S variable, like this: [global] log file = %S.log 12.1.1.1 Log levels The level of logging that Samba uses can be set in the smb.conf file using the global log level or debug level option; they are equivalent. The logging level is an integer that can range from 0 to 10. At level 0, no logging is done. Higher values result in more voluminous logging. For example, let's assume that we will use a Windows client to browse a directory on a Samba server. For a small amount of log information, you can use log level = 1 , which instructs Samba to show only cursory information, in this case only the connection itself: 05/25/02 22:02:11 server (192.168.236.86) connect to service public as user pcguest (uid=503,gid=100) (pid 3377) Higher debug levels produce more detailed information. Usually, you won't need more than level 3, which is fully adequate for most Samba administrators. Levels above 3 are used by the developers and dump enormous amounts of cryptic information. Here is an example of output at levels 2 and 3 for the same operation. Don't worry if you don't understand the intricacies of an SMB connection; the point is simply to show you what types of information are shown at the different logging levels: /* Level 2 */ Got SIGHUP Processing section "[homes]" Processing section "[public]" Processing section "[temp]" Allowed connection from 192.168.236.86 (192.168.236.86) to IPC$ Allowed connection from 192.168.236.86 (192.168.236.86) to IPC/ /* Level 3 */ 05/25/02 22:15:09 Transaction 63 of length 67 switch message SMBtconX (pid 3377) Allowed connection from 192.168.236.86 (192.168.236.86) to IPC$ ACCEPTED: guest account and guest ok found free connection number 105 Connect path is /tmp chdir to /tmp chdir to / 05/25/02 22:15:09 server (192.168.236.86) connect to service IPC$ as user pcguest (uid=503,gid=100) (pid 3377) 05/25/02 22:15:09 tconX service=ipc$ user=pcguest cnum=105 05/25/02 22:15:09 Transaction 64 of length 99 switch message SMBtrans (pid 3377) chdir to /tmp trans <\PIPE\LANMAN> data=0 params=19 setup=0 Got API command 0 of form <WrLeh> <B13BWz> (tdscnt=0,tpscnt=19,mdrcnt=4096,mprcnt=8) Doing RNetShareEnum RNetShareEnum gave 4 entries of 4 (1 4096 126 4096) 05/25/02 22:15:11 Transaction 65 of length 99 switch message SMBtrans (pid 3377) chdir to / chdir to /tmp trans <\PIPE\LANMAN> data=0 params=19 setup=0 Got API command 0 of form <WrLeh> <B13BWz> (tdscnt=0,tpscnt=19,mdrcnt=4096,mprcnt=8) Doing RNetShareEnum RNetShareEnum gave 4 entries of 4 (1 4096 126 4096) 05/25/02 22:15:11 Transaction 66 of length 95 switch message SMBtrans2 (pid 3377) chdir to / chdir to /pcdisk/public call_trans2findfirst: dirtype = 0, maxentries = 6, close_after_first=0, close_if_end = 0 requires_resume_key = 0 level = 260, max_data_bytes = 2432 unix_clean_name [./DESKTOP.INI] unix_clean_name [desktop.ini] unix_clean_name [./] creating new dirptr 1 for path ./, expect_close = 1 05/25/02 22:15:11 Transaction 67 of length 53 switch message SMBgetatr (pid 3377) chdir to / [... deleted ...] We cut off this listing after the first packet because it runs on for many pages. However, be aware that log levels above 3 will quickly consume disk space with megabytes of excruciating detail concerning Samba's internal operations. Log level 3 is extremely useful for following exactly what the server is doing, and most of the time it will be obvious where an error occurs by glancing through the log file. Using a high log level (3 or above) will seriously slow down the Samba server. Remember that every log message generated causes a write to disk (an inherently slow operation) and log levels greater than 2 produce massive amounts of data. Essentially , you should turn on logging level 3 only when you're actively tracking a problem in the Samba server. 12.1.1.2 Activating and deactivating logging To turn logging on and off, set the appropriate level in the [global] section of smb.conf . Then, you can either restart Samba or force the current daemon to reprocess the configuration file by sending it a hangup (HUP) signal. You also can send the smbd process a SIGUSR1 signal to increase its log level by one while it's running, like this: # kill -SIGUSR1 1234 or a SIGUSR2 signal to decrease it by one: # kill -SIGUSR2 1234 12.1.1.3 Logging by individual client systems or users An effective way to diagnose problems without hampering other users is to assign different log levels for different systems in the [global] section of the smb.conf file. We can do this by building on the strategy we presented earlier: [global] log level = 0 log file = /usr/local/samba/var/log.%m include = /usr/local/samba/lib/smb.conf.%m These options instruct Samba to use unique configuration and log files for each client that connects. Now all you have to do is create an smb.conf file for a specific client system with a log level = 3 entry in it (the others will pick up the default log level of 0) and use that log file to track down the problem. Similarly, if only particular users are experiencing a problem ”and it travels from system to system with them ”you can isolate logging to a specific user by adding the following to the smb.conf file: [global] log level = 0 log file = /usr/local/samba/var/log.%u include = /usr/local/samba/lib/smb.conf.%u Then you can create a unique smb.conf file for each user you wish to monitor (e.g., /usr/local/samba/lib/smb.conf.tim ). Files containing the configuration option log level = 3 and only those users will get more detailed logging. 12.1.2 Samba Test Utilities A rigorous set of tests that exercise the major parts of Samba are described in various files in the /docs/textdocs directory of the Samba distribution kit, starting with DIAGNOSIS.txt . The fault tree in this chapter is a more detailed version of the basic tests suggested by the Samba Team, but it covers only installation and reconfiguration diagnosis, such as DIAGNOSIS.txt . The other files in the /docs subdirectories address specific problems and instruct you how to troubleshoot items not included in this book. If the fault tree doesn't suffice, be sure to look at DIAGNOSIS.txt and its friends . 12.1.3 Unix Utilities Sometimes it's useful to use a tool outside the Samba suite to examine what's happening inside the server. Three diagnostic tools can be of particular help in debugging Samba troubles: trace , tcpdump , and Ethereal . 12.1.3.1 Using trace The trace command masquerades under several different names , depending on the operating system you are using. On Linux it will be strace ; on Solaris you'll use truss ; SGI will have padc and par ; and HP-UX will have trace or tusc . All have essentially the same function, which is to display each operating system function call as it is executed. This allows you to follow the execution of a program, such as the Samba server, and often pinpoints the exact call that is causing the difficulty. One problem that trace can highlight is an incorrect version of a dynamically linked library. This can happen if you've downloaded prebuilt binaries of Samba. You'll typically see the offending call at the end of the trace , just before the program terminates. A sample strace output for the Linux operating system follows . This is a small section of a larger file created during the opening of a directory on the Samba server. Each line lists a system call and includes its parameters and the return value. If there was an error, the error value (e.g., ENOENT ) and its explanation are also shown. You can look up the parameter types and the errors that can occur in the appropriate trace manual page for the operating system you are using. chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat(5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek(5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, " chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 #7SMB chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 "..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, " chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 ?", 4) = 4 read(3, "7SMBu chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 chdir("/pcdisk/public") = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) stat("mini", {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 stat("mini/desktop.ini", 0xbffff7ec) = -1 ENOENT (No such file or directory) open ("mini", O_RDONLY) = 5 fcntl(5, F_SETFD, FD_CLOEXEC) = 0 fstat (5, {st_mode=S_IFDIR0755, st_size=1024, ...}) = 0 lseek(5, 0, SEEK_CUR) = 0 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 196 lseek (5, 0, SEEK_CUR) = 1024 SYS_141(0x5, 0xbfffdbbc, 0xedc, 0xbfffdbbc, 0x80ba708) = 0 close(5) = 0 stat("mini/desktop.ini", 0xbffff86c) = -1 ENOENT (No such file or directory) write(3, "\0\0\0#\377SMB\10\1\0\2\0\200\1\0"..., 39) = 39 SYS_142(0xff, 0xbffffc3c, 0, 0, 0xbffffc08) = 1 read(3, "\0\0\0?", 4) = 4 read(3, "\377SMBu\0\0\0\0\0\0\0\0\0\0\0\0"..., 63) = 63 time(NULL) = 896143871 "..., 63) = 63 time(NULL) = 896143871 This example shows several stat() calls failing to find the files they were expecting. You don't have to be an expert to see that the file desktop.ini is missing from that directory. In fact, many difficult problems can be identified by looking for obvious, repeatable errors with trace . Often, you need not look further than the last message before a crash. 12.1.3.2 Using tcpdump The tcpdump program, as extended by Andrew Tridgell, allows you to monitor SMB network traffic in real time. A variety of output formats are available, and you can filter the output to look at only a particular type of traffic. You can examine all conversations between client and server, including SMB and NMB broadcast messages. While its troubleshooting capabilities lie mainly at the OSI network layer, you can still use its output to get a general idea of what the server and client are attempting to do. A sample tcpdump log follows. In this instance, the client has requested a directory listing, and the server has responded appropriately, giving the directory names homes , public , IPC$ , and temp (we've added a few explanations on the right): $ tcpdump -v -s 255 -i eth0 port not telnet SMB PACKET: SMBtrans (REQUEST) Request packet SMB Command = 0x25 Request was ls or dir [000] 01 00 00 10 .... >>> NBT Packet Outer frame of SMB packet NBT Session Packet Flags=0x0 Length=226 [lines skipped] SMB PACKET: SMBtrans (REPLY) Beginning of a reply to request SMB Command = 0x25 Command was an ls or dir Error class = 0x0 Error code = 0 No errors Flags1 = 0x80 Flags2 = 0x1 Tree ID = 105 Proc ID = 6075 UID = 100 MID = 30337 Word Count = 10 TotParamCnt=8 TotDataCnt=163 Res1=0 ParamCnt=8 ParamOff=55 Res2=0 DataCnt=163 DataOff=63 Res3=0 Lsetup=0 Param Data: (8 bytes) [000] 00 00 00 00 05 00 05 00 ........ Data Data: (135 bytes) Actual directory contents: [000] 68 6F 6D 65 73 00 00 00 00 00 00 00 00 00 00 00 homes... ........ [010] 64 00 00 00 70 75 62 6C 69 63 00 00 00 00 00 00 d...publ ic...... [020] 00 00 00 00 75 00 00 00 74 65 6D 70 00 00 00 00 ....u... temp.... [030] 00 00 00 00 00 00 00 00 76 00 00 00 49 50 43 24 ........ v...IPC$ [040] 00 00 00 00 00 00 00 00 00 00 03 00 77 00 00 00 ........ ....w... [050] 64 6F 6E 68 61 6D 00 00 00 00 00 00 00 00 00 00 donham.. ........ [060] 92 00 00 00 48 6F 6D 65 20 44 69 72 65 63 74 6F ....Home Directo [070] 72 69 65 73 00 00 00 49 50 43 20 53 65 72 76 69 ries...I PC Servi [080] 63 65 20 28 53 61 6D ce (Sam This is more of the same debugging session as we saw before with the trace command: the listing of a directory. The options we used were -v (verbose), -i eth0 to tell tcpdump on which interface to listen (an Ethernet port), and -s 255 to tell it to save the first 255 bytes of each packet instead of the default: the first 68. The option port not telnet is used to avoid screens of telnet traffic, because we were logged in to the server remotely. The tcpdump program actually has quite a number of options to filter just the traffic you want to look at. If you've used snoop or etherdump , it will look vaguely familiar. You can download the modified tcpdump from the Samba FTP server, located at ftp://samba.anu.edu.au/pub/samba/tcpdump-smb. Other versions might not include support for the SMB protocol; if you don't see output such as that shown in the example, you'll need to use the SMB-enabled version. 12.1.3.3 Using Ethereal Ethereal (http://www.ethereal.com) is a GUI-based utility that performs the same basic function as tcpdump . You might prefer Ethereal because it is much easier to use. Once you have Ethereal running, just do the following: -
Select Start from the Capture menu. -
Click the OK button in the dialog box that appears. This will bring up a dialog box showing how many packets Ethereal has seen. Perform the actions on the system(s) in your network to reproduce the problem you are analyzing. -
Click the Stop button in the Ethereal dialog box to make it finish collecting data. -
In the main Ethereal window, click any item in the upper window to view it in the lower window. In the lower window, click any of the boxes containing a plus sign ( + ) to expand the view. Ethereal does a good job of translating the content of the packets it encounters into human-readable format, and you should have little trouble seeing what happened on the network during the capture period. | 