Step 5.1 General Troubleshooting

The primary purpose of this guide is to provide basic, step-by-step instructions for quickly installing, configuring, and using OpenSSH. Of course, as with most new ventures , you will most likely run into unexpected problems along the way. OpenSSH runs on numerous platforms so of course it would be an incredible, if not impossible , task to address all known problems encountered on every platform. This section was never intended to address every known problem but rather is intended to be a guide to help with general troubleshooting of SSH issues and also to address specific common problems.

You may have other troubleshooting techniques to share or may learn of solutions to new problems you encounter “ if you would like to share your findings, you are encouraged to do so via e-mail to [e-mail].

Action 5 1 1 Verify download and installation configuration of software

If you are encountering problems just getting OpenSSH to run, and/or are receiving error messages that indicate an install or configure problem, then of course the first place to start is to review the installation/configuration steps for the OpenSSH pre-requisite software and for OpenSSH itself (see Section I). Remember that certain software - such as Zlib and OpenSSL - must be installed before OpenSSH, since OpenSSH relies on them.

Before you get into specific install reviews, it's a good idea to make sure that you have a valid download file. It might appear obvious that you don't have a download problem if you were able to "unpack", install and compile the source code “ but stranger things have happened . Make sure that the size of the downloaded file (typically .tar, .Z, .gz, .zip) is equal to that provided (usually) by the download site. If you performed an FTP download, check the site to determine if it was necessary to perform a BIN (binary) transfer ” and if so, make sure that you did so (To perform a binary transfer, enter "BIN" at the "ftp>" prompt). You should also verify any checksums provided for downloaded files.

To ensure that the OpenSSH content is complete and has not been corrupted in transit or maliciously modified, the PGP signature should also be checked. Note that a trojaned version of OpenSSH has previously been placed on hacked mirror FTP sites (See CERT CA-2002-24 for details). Unfortunately, the where- abouts of the OpenSSH public key are not well publicized at the time of this writing. The public key currently used to sign OpenSSH belongs to Damien Miller (Personal Key) and may be retrieved from PGP key servers using id 86FF9C48. The following example commands use the open source GNU Privacy Guard (GPG) from http://www.gnupg.org/.

The key is also available from the mirror FTP sites, however, if you retrieve it from the same FTP server from which you downloaded the software, you run the risk that both have been maliciously altered . Once the key is imported, the tar bundle can be verified . It is not necessary to assign any trust to the key.

$> gpg --keyserver wwwkeys.pgp.net --recv-keys 86FF9C48

 $> gpg --list-keys 86FF9C48
 pub 1024D/86FF9C48 2001-02-26 Damien Miller (Personal Key) 
 sub 2048g/AA2B1C41 2001-02-26
 $> gpg --verify openssh-3.5p1.tar.gz.sig openssh-3.5p1.tar.gz
 gpg: Signature made Thu Oct 3 21:34:43 2002 EDT using DSA key ID 86FF9C48
 gpg: Good signature from "Damien Miller (Personal Key) "

 Could not find a valid trust path to the key. Let's see whether we can assign some missing owner
 trust values.

 No path leading to one of our keys found.

 gpg: WARNING: This key is not certified with a trusted signature!
 gpg: There is no indication that the signature belongs to the owner.
 gpg: Fingerprint: 3981 992A 1523 ABA0 79DB FC66 CE8E CB03 86FF 9C48

Many open-source tools provide a significant amount of verbose output when they are installed, so it can be easy to overlook error messages. Therefore, it's a good idea to review any install messages/logs for the pre-requisite software if you still have them. If these are not available, then you may still be able to validate the installations “ most of the open-source tools come with helpful installation/configuration documentation. Check for INSTALL and README files within the directories where you stored the source code for each package. These files frequently provide good detail and can be of great help. For example, notice the INSTALL and README files that are provided with the OpenSSL distribution:

# pwd
 /usr/local/openssl-0.9.6g
 # ls
 CHANGES MacOS apps include perl
 CHANGES.SSLeay Makefile bugs install.com rehash.time
 Configure Makefile.org certs libRSAglue.a rsaref
 FAQ Makefile.ssl config libcrypto.a shlib
 INSTALL NEWS crypto libssl.a ssl
 INSTALL.MacOS PROBLEMS demos makevms.com test
 INSTALL.VMS README doc ms times
 INSTALL.W32 README.ENGINE e_os.h openssl.doxy tools
 LICENSE VMS e_os2.h openssl.spec util

Action 5 1 2 Verify OpenSSH commands are in the UNIX search path

If you get a "not found" error when trying to invoke one of the OpenSSH executables (ssh, slogin , scp, sftp, etc), then make sure that the UNIX PATH environment variable includes the location of the executables; otherwise the OpenSSH commands' location will have to be provided when the executables are invoked. For example:

$ ssh server.example.com
 ssh: not found
 $ echo $PATH
 /usr/sbin:/usr/bin:/usr/dt/bin:/usr/openwin/bin:/bin:/usr/ucb:/usr/ccs/bin

To update the search PATH, first verify the location of the executables. For example, if you installed the executables in the /usr/local/bin directory, the following commands in a Bourne-, Korn-, or Z-Shell will ensure that this directory will be searched:

$ PATH=$PATH:/usr/local/bin; export PATH
 $ echo $PATH
 /usr/sbin:/usr/bin:/usr/dt/bin:/usr/openwin/bin:/bin:/usr/ucb:/usr/ccs/bin:/usr/local/bin

A subsequent SSH attempt is now successful:

$ ssh server.example.com
 [e-mail]'s password: *******
 Last login: Mon Feb 17 14:05:03 2003 from client.example.com
 $

Action 5 1 3 Utilize verbose option to OpenSSH commands

Sometimes the “v verbose option available with most of the OpenSSH commands can provide insight. This option provides much more detail than is normally received. For example, here is the previous connection attempt using the “v flag:

$ ssh -v server.example.com
 OpenSSH_3.5p1, SSH protocols 1.5/2.0, OpenSSL 0x0090607f
 debug1: Reading configuration data /usr/local/etc/ssh_config
 debug1: Rhosts Authentication disabled, originating port will not be trusted.
 debug1: ssh_connect: needpriv 0
 debug1: Connecting to server.example.com [172.16.121.6] port 22.
 debug1: Connection established.
 debug1: identity file /home/sshuser/.ssh/id_rsa type 1
 debug1: Remote protocol version 2.0, remote software version OpenSSH_3.4p1
 debug1: match: OpenSSH_3.4p1 pat OpenSSH*
 debug1: Enabling compatibility mode for protocol 2.0
 debug1: Local version string SSH-2.0-OpenSSH_3.5p1
 debug1: SSH2_MSG_KEXINIT sent
 debug1: SSH2_MSG_KEXINIT received
 debug1: kex: server->client aes128-cbc hmac-md5 none
 debug1: kex: client->server aes128-cbc hmac-md5 none
 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
 debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
 debug1: dh_gen_key: priv key bits set: 113/256
 debug1: bits set: 1602/3191
 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
 debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
 debug1: Host 'server.example.com' is known and matches the RSA host key.
 debug1: Found key in /home/sshuser/.ssh/known_hosts:1
 debug1: bits set: 1595/3191
 debug1: ssh_rsa_verify: signature correct
 debug1: kex_derive_keys
		debug1: newkeys: mode
 debug1: SSH2_MSG_NEWKEYS sent
 debug1: waiting for SSH2_MSG_NEWKEYS
 debug1: newkeys: mode 0
 debug1: SSH2_MSG_NEWKEYS received
 debug1: done: ssh_kex2.
 debug1: send SSH2_MSG_SERVICE_REQUEST
 debug1: service_accept: ssh-userauth
 debug1: got SSH2_MSG_SERVICE_ACCEPT
 debug1: authentications that can continue: publickey,password,keyboard-interactive
 debug1: next auth method to try is publickey
 debug1: try pubkey: /export/home/sshuser/.ssh/id_rsa
 debug1: authentications that can continue: publickey,password,keyboard-interactive
 debug1: next auth method to try is keyboard-interactive
 debug1: authentications that can continue: publickey,password,keyboard-interactive
 debug1: next auth method to try is password
 [e-mail]'s password

As you can see, the verbose flag provides helpful debugging information - in this example, the client " knows " about the server (i.e. this is not the first SSH connection attempt), the client and server have "negotiated" to use the SSHv2 protocol, and public key authentication has failed (possibly this type of authentication has not yet been configured or the permissions for the file containing the private key have not been set to a secure level), so the UNIX password authentication is tried next (default behavior for SSH authentication).

Tech Tip  

You should note that multiple -v options are supported - e.g. ssh -vv server.example.com . For each additional “v provided, more verbose output will be provided.

Action 5 1 4 Utilize OpenSSH server daemon log

By default, the OpenSSH server daemon (sshd) logs certain information, such as authentication failures, to the UNIX syslog facility. These messages can sometimes provide insight, especially if the problem is just an incorrect password or passphrase. For example, here is an entry indicating an authentication failure on an OpenSSH server:

Feb 19 13:37:50 server.example.com sshd[10024]: [ID 800047 auth.crit] fatal: Timeout before
 authentication for 172.16.121.16.

You can get much more verbose output in the OpenSSH server log file by restarting the sshd daemon with the -d (debug) command line option. For example, on the OpenSSH server:

# /usr/local/sbin/sshd -d &
 [1] 14027

If you use the “d flag, be prepared to wade through a lot of detail. The following is the verbose output from a client connection whereby the public key authentication failed and therefore UNIX password authentication was used (which is the default sshd behavior):

debug1: sshd version OpenSSH_3.4p1
 debug1: read PEM private key done: type RSA
 debug1: private host key: #0 type 1 RSA
 debug1: read PEM private key done: type DSA
 debug1: private host key: #1 type 2 DSA
 debug1: Bind to port 22 on ::.
 Server listening on :: port 22.
 debug1: Bind to port 22 on 0.0.0.0.
 Server listening on 0.0.0.0 port 22.
 debug1: Server will not fork when running in debugging mode.
 Connection from 172.16.121.16 port 33672
 debug1: Client protocol version 2.0; client software version OpenSSH_3.5p1
 debug1: match: OpenSSH_3.5p1 pat OpenSSH*
 Enabling compatibility mode for protocol 2.0
 debug1: Local version string SSH-2.0-OpenSSH_3.4p1
 debug1: list_hostkey_types: ssh-rsa,ssh-dss
 debug1: SSH2_MSG_KEXINIT sent
		debug1: SSH2_MSG_KEXINIT received
 debug1: kex: client->server aes128-cbc hmac-md5 none
 debug1: kex: server->client aes128-cbc hmac-md5 none
 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
 debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
 debug1: dh_gen_key: priv key bits set: 133/256
 debug1: bits set: 1590/3191
 debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
 debug1: bits set: 1617/3191
 debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
 debug1: kex_derive_keys
 debug1: newkeys: mode 1
 debug1: SSH2_MSG_NEWKEYS sent
 debug1: waiting for SSH2_MSG_NEWKEYS
 debug1: newkeys: mode 0
 debug1: SSH2_MSG_NEWKEYS received
 debug1: KEX done
 debug1: userauth-request for user sshuser service ssh-connection method none
 debug1: attempt 0 failures 0
 Failed none for sshuser from 172.16.121.16 port 33672 ssh2
 Failed none for sshuser from 172.16.121.16 port 33672 ssh2
 debug1: userauth-request for user sshuser service ssh-connection method publickey
 debug1: attempt 1 failures 1
		debug1: test whether pkalg/pkblob are acceptable
 debug1: temporarily_use_uid: 1009/150 (e=0)
 debug1: trying public key file /home/sshuser/.ssh/authorized_keys
 debug1: matching key found: file /home/sshuser/.ssh/authorized_keys, line 1
 Found matching RSA key: a4:93:99:99:29:9f:41:9d:af:21:54:b6:8f:3e:ce:c6
 debug1: restore_uidPostponed publickey for sshuser from 172.16.121.16 port 33672 ssh2
 debug1: userauth-request for user sshuser service ssh-connection method keyboard-interactive
 debug1: attempt 2 failures 1
 debug1: keyboard-interactive devs
 debug1: auth2_challenge: user=sshuser devs=
 debug1: kbdint_alloc: devices "Failed keyboard-interactive for sshuser from
 172.16.121.16 port 33672 ssh2
 debug1: userauth-request for user sshuser service ssh-connection method password
 debug1: attempt 3 failures 2Failed password for sshuser from 172.16.121.16 port 33672 ssh2Failed password for sshuser from
 172.16.121.16 port 33672 ssh2
 debug1: userauth-request for user sshuser service ssh-connection method password
 debug1: attempt 4 failures 3
 Accepted password for sshuser from 172.16.121.16 port 33672 ssh2
 debug1: monitor_child_preauth: sshuser has been authenticated by privileged process
 Accepted password for sshuser from 172.16.121.16 port 33672 ssh2
 debug1: newkeys: mode 0
 debug1: newkeys: mode 1
 debug1: Entering interactive session for SSH2.
 debug1: fd 4 setting O_NONBLOCK
 debug1: fd 5 setting O_NONBLOCK
 debug1: server_init_dispatch_20
 debug1: server_input_channel_open: ctype session rchan 0 win 65536 max 16384
 debug1: input_session_request
 debug1: channel 0: new [server-session]
 debug1: session_new: init
 debug1: session_new: session 0
 debug1: session_open: channel 0
 debug1: session_open: session 0: link with channel 0
 debug1: server_input_channel_open: confirm session
 debug1: server_input_channel_req: channel 0 request pty-req reply 0
 debug1: session_by_channel: session 0 channel 0
 debug1: session_input_channel_req: session 0 req pty-req
 debug1: Allocating pty.
 debug1: session_new: init
 debug1: session_new: session 0
 debug1: session_pty_req: session 0 alloc /dev/pts/22
 debug1: server_input_channel_req: channel 0 request x11-req reply 0
 debug1: session_by_channel: session 0 channel 0
 debug1: session_input_channel_req: session 0 req x11-req
 debug1: fd 12 setting O_NONBLOCK
 debug1: channel 1: new [X11 inet listener]
 debug1: fd 13 setting O_NONBLOCK
 debug1: channel 2: new [X11 inet listener]
 debug1: server_input_channel_req: channel 0 request shell reply 0
 debug1: session_by_channel: session 0 channel 0
 debug1: session_input_channel_req: session 0 req shell

Tech Tip  

Multiple -d options can be utilized when launching the OpenSSH daemon - e.g. usr/local/sbin/sshd -d -d & to provide an additional level of debugging detail.

Action 5 1 5 Run the OpenSSH Server in foreground verbose mode

Running the server in foreground debug mode ( “D “d ) is very useful for debugging a wide variety of connection problems. When run in the foreground, the server will need to be restarted for each connection attempt:

# /usr/local/sbin/sshd -D -d

 debug1: Seeding random number generator
 debug1: sshd version OpenSSH_3.0.1p1
 debug1: private host key: #0 type 0 RSA1
 debug1: read PEM private key done: type RSA
 debug1: private host key: #1 type 1 RSA
 debug1: read PEM private key done: type DSA
 debug1: private host key: #2 type 2 DSA
 debug1: Bind to port 22 on 192.168.102.12.
 Server listening on 192.168.102.12 port 22.
 debug1: Bind to port 22 on 216.55.177.197.
 Server listening on 192.168.102.12 port 22.
 debug1: Server will not fork when running in debugging mode.
 Connection from 10.10.102.84 port 1446
 debug1: Client protocol version 2.0; client software version PuTTY-Release-0.53b
 debug1: no match: PuTTY-Release-0.53b
 Enabling compatibility mode for protocol 2.0
 debug1: Local version string SSH-2.0-OpenSSH_3.0.1p1
 debug1: Rhosts Authentication disabled, originating port 1446 not trusted.
 debug1: list_hostkey_types: ssh-rsa,ssh-dss
 debug1: SSH2_MSG_KEXINIT sent
 debug1: SSH2_MSG_KEXINIT received
 debug1: kex: client->server 3des-cbc hmac-sha1 none
 debug1: kex: server->client 3des-cbc hmac-sha1 none
 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST_OLD received
 debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
 debug1: dh_gen_key: priv key bits set: 193/384
 debug1: bits set: 1595/3191
 debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
 debug1: bits set: 1599/3191
 debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
 debug1: kex_derive_keys
 debug1: newkeys: mode 1
 debug1: SSH2_MSG_NEWKEYS sent
 debug1: waiting for SSH2_MSG_NEWKEYS
 debug1: newkeys: mode 0
 debug1: SSH2_MSG_NEWKEYS received
 debug1: KEX done