Linux-PAM is the Linux implementation of PAM. HPUX, Solaris, and other operation systems use PAM. Linux-PAM is a package designed to handle user authentication, which means each application does not need to include a routine to ask for a password, encrypt it with crypt(2), and compare it against the entry in /etc/shadow. PAM does much more than just verifying passwords, however. The access rules can be changed by editing a single PAM configuration file. PAM is a very important part of Linux security. Applications must be written to use PAM.
The heart of PAM is the configuration directory /etc/pam.d or configuration file /etc/pam.conf for older implementations of PAM.
A PAM-enabled application uses a configuration file in /etc/pam.d or a block of lines in /etc/pam.conf. Let's look at /etc/pam.d first because it is the newer and more common method. PAM looks for a configuration file with the name of the application. If one doesn't exist, the file other is used. The login file in /etc/pam.d shows the PAM modules used for the login application. The following is a sample /etc/pam.d/login file from a Red Hat Enterprise Linux AS release 3 system:
#%PAM-1.0 auth required pam_securetty.so auth required pam_stack.so service=system-auth auth required pam_nologin.so account required pam_stack.so service=system-auth password required pam_stack.so service=system-auth session required pam_stack.so service=system-auth session optional pam_console.so
The files in /etc/pam.d have four fields: function, priority, module name, and module arguments. The group of modules in a PAM configuration file is called a module stack. Linux distributions have different authorization modules, but they follow the same format.
Four functions can be configured: account, authentication, password, and session (see Table 14-7). As you can see from the login example, all four can be used by the same application.
The priority field determines what happens if the module fails to grant the function. This field has four allowable values (see Table 14-8).
This is the full path of the module to be used or the name of a file in /lib/security.
Some modules take arguments. These are documented in the module README files. The location of the PAM README files varies, but the files are probably in a subdirectory of /usr/share/doc/. Use rpm -q -filesbypkg pam_ if you are having trouble finding the README files.
The following is from a Red Hat 3.0ES system, and it demonstrates how many applications rely on PAM to perform authorization tasks.
# ls -ld /etc/pam.d drwxr-xr-x 2 root root 4096 Oct 30 04:25 /etc/pam.d # ls /etc/pam.d authconfig printconf-tui redhat-config-users authconfig-gtk printtool redhat-config-xfree86 bindconf reboot redhat-install-packages chfn redhat-cdinstall-helper redhat-logviewer chsh redhat-config-authentication redhat-switch-mail cups redhat-config-bind redhat-switch-mail-nox dateconfig redhat-config-date rhn_register ethereal redhat-config-httpd samba gdm redhat-config-keyboard screen gdm-autologin redhat-config-language serviceconf gdmsetup redhat-config-mouse setup halt redhat-config-netboot smtp hwbrowser redhat-config-network smtp.sendmail imap redhat-config-network-cmd sshd internet-druid redhat-config-network-druid su kbdrate redhat-config-nfs sudo kde redhat-config-packages synaptic kppp redhat-config-printer system-auth login redhat-config-printer-gui up2date neat redhat-config-printer-tui up2date-config other redhat-config-proc up2date-nox passwd redhat-config-rootpassword webmin pop redhat-config-samba xdm poweroff redhat-config-securitylevel xscreensaver ppp redhat-config-services xserver printconf redhat-config-soundcard printconf-gui redhat-config-time
If /etc/pam.d exists, /etc/pam.conf is ignored. The /etc/pam.conf file lines are identical to those of /etc/pam.d except that they contain one extra field. The lines in /etc/pam.conf contain the session name as the first field. The session name identifies the stack to which a line belongs.
The libraries supporting the /etc/pam.d authorization modules all have 755 permissions and are owned by root with group root. The following is from a Red Hat 9.0 system:
#ls -ld /lib/security drwxr-xr-x 3 root root 4096 May 18 2004 /lib/security #ls -l /lib/security total 1276 -rwxr-xr-x 1 root root 9696 Feb 10 2003 pam_access.so -rwxr-xr-x 1 root root 6320 Feb 10 2003 pam_chroot.so -rwxr-xr-x 1 root root 47584 Feb 10 2003 pam_console.so -rwxr-xr-x 1 root root 12964 Feb 10 2003 pam_cracklib.so -rwxr-xr-x 1 root root 3404 Feb 10 2003 pam_deny.so -rwxr-xr-x 1 root root 11592 Feb 10 2003 pam_env.so drwxr-xr-x 2 root root 4096 Nov 6 2003 pam_filter -rwxr-xr-x 1 root root 11208 Feb 10 2003 pam_filter.so -rwxr-xr-x 1 root root 6048 Feb 10 2003 pam_ftp.so -rwxr-xr-x 1 root root 11148 Feb 10 2003 pam_group.so -rwxr-xr-x 1 root root 7540 Feb 10 2003 pam_issue.so -rwxr-xr-x 1 root root 59508 Jan 30 2003 pam_krb5afs.so -rwxr-xr-x 1 root root 57464 Jan 30 2003 pam_krb5.so -rwxr-xr-x 1 root root 8468 Feb 10 2003 pam_lastlog.so -rwxr-xr-x 1 root root 39080 Jan 25 2003 pam_ldap.so -rwxr-xr-x 1 root root 12324 Feb 10 2003 pam_limits.so -rwxr-xr-x 1 root root 10740 Feb 10 2003 pam_listfile.so -rwxr-xr-x 1 root root 9620 Feb 10 2003 pam_localuser.so -rwxr-xr-x 1 root root 9664 Feb 10 2003 pam_mail.so -rwxr-xr-x 1 root root 16652 Feb 10 2003 pam_mkhomedir.so -rwxr-xr-x 1 root root 4272 Feb 10 2003 pam_motd.so -rwxr-xr-x 1 root root 4856 Feb 10 2003 pam_nologin.so -rwxr-xr-x 1 root root 3708 Feb 10 2003 pam_permit.so -rwxr-xr-x 1 root root 46336 Feb 10 2003 pam_pwdb.so -rwxr-xr-x 1 root root 11372 Feb 10 2003 pam_rhosts_auth.so -rwxr-xr-x 1 root root 3936 Feb 10 2003 pam_rootok.so -rwxr-xr-x 1 root root 6544 Feb 10 2003 pam_securetty.so -rwxr-xr-x 1 root root 5520 Feb 10 2003 pam_shells.so -rwxr-xr-x 1 root root 43588 Aug 15 2003 pam_smb_auth.so -rwxr-xr-x 1 root root 468260 Apr 5 2003 pam_smbpass.so -rwxr-xr-x 1 root root 11132 Feb 10 2003 pam_stack.so -rwxr-xr-x 1 root root 10676 Feb 10 2003 pam_stress.so -rwxr-xr-x 1 root root 13752 Feb 10 2003 pam_tally.so -rwxr-xr-x 1 root root 9752 Feb 10 2003 pam_time.so -rwxr-xr-x 1 root root 10544 Feb 10 2003 pam_timestamp.so lrwxrwxrwx 1 root root 11 Nov 6 2003 pam_unix_acct.so -> pam_unix.so lrwxrwxrwx 1 root root 11 Nov 6 2003 pam_unix_auth.so -> pam_unix.so lrwxrwxrwx 1 root root 11 Nov 6 2003 pam_unix_passwd.so -> pam_unix.so lrwxrwxrwx 1 root root 11 Nov 6 2003 pam_unix_session.so -> pam_unix.so -rwxr-xr-x 1 root root 48544 Feb 10 2003 pam_unix.so -rwxr-xr-x 1 root root 9148 Feb 10 2003 pam_userdb.so -rwxr-xr-x 1 root root 4644 Feb 10 2003 pam_warn.so -rwxr-xr-x 1 root root 7788 Feb 10 2003 pam_wheel.so -rwxr-xr-x 1 root root 13348 Apr 5 2003 pam_winbind.so -rwxr-xr-x 1 root root 13860 Feb 10 2003 pam_xauth.so
The previous coverage is just a brief explanation of Linux-PAM that is needed before we start looking at troubleshooting PAM problems. Detailed PAM information can be found in The System Administrators' Guide, The Module Writers' Manual, and The Application Developers' Manual available from the Linux-PAM home page (http://www.kernel.org/pub/linux/libs/pam/) under the "online documentation" link. There is also a PAM man page and documentation in /usr/share/doc. The pam(8) man page is excellent as well.
The lab instructor for my first college programming class gave us good advice. He said that we would have bugs in our code that we could not find. We would look very hard and still not find the cause. He said eventually we would decide the problem was caused by a hardware or compiler issue. He said we would be wrong. The point of his lecture was that yes, problems with compilers sometimes occur, and CPUs and memory chips sometimes fail, but it was more likely that our code was buggy. He did not want a bunch of freshman annoying the IT department.
There certainly might be bugs in PAM or a PAM module. Before going there, however, please look over the following questions to confirm that PAM is installed properly. Use rpm to verify all PAM packages, modules, libraries, and so on. If a module has been customized, put the default version back and test with it. The same goes for the application using PAM. The problem may be with sshd or login and not PAM.
Scenario 14-1: Missing Stack Prevents Login
In the next example, the /etc/pam.d/login module is missing, and no one can log in.
Red Hat Linux release 9 (Shrike) Kernel 2.4.20-8 on an i686 sawnee.somecomp.com login: root Login incorrect Login incorrect Login incorrect Login incorrect Red Hat Linux release 9 (Shrike) Kernel 2.4.20-8 on an i686 sawnee.somecomp.com login:
The user never even gets an opportunity to enter the password. Because there is no /etc/pam.d/login module, the default module, /etc/pam.d/other, is used. The other file denies authorization outright:
#cat /etc/pam.d/other #%PAM-1.0 auth required /lib/security/$ISA/pam_deny.so account required /lib/security/$ISA/pam_deny.so password required /lib/security/$ISA/pam_deny.so session required /lib/security/$ISA/pam_deny.so
What happens if other is missing too?
Red Hat Linux release 9 (Shrike) Kernel 2.4.20-8 on an i686 sawnee.somecomp.com login: root login: PAM Failure, aborting: Critical error - immediate abort Red Hat Linux release 9 (Shrike) Kernel 2.4.20-8 on an i686 sawnee.somecomp.com login:
PAM does not grant authorization if there is no authorization module. The solution is to create the /etc/pam.d/login file or restore it from backup. If there is an open root session, we can make the fix. If there is none, the box must be booted to single user mode and the fix made from there. Refer to Chapter 1, "System Boot, Startup, and Shutdown Issues," if this subject is not familiar.
Scenario 14-2: Missing /etc/pam.d Prevents Login
A user tries to connect with ssh, but the window closes immediately after the user enters the user name. Again, no one can log in. Login at the console yields the following:
sawnee.somecomp.com login: dave login: PAM Failure, aborting: Critical error - immediate abort
This problem has symptoms similar to the missing /etc/pam.d/login example. If there were an open root session, we could troubleshoot PAM. Because there are none, the box must be booted to single user mode. In single user mode, we can quickly see that there is no /etc/pam.d directory. After reading this chapter, the solution should be pretty obvious. We need to create the pam.d directory and one module for system access:
#cd /etc #mkdir pam.d #chmod 755 pam.d #cd pam.d
Create a file called login with the following entries:
auth required pam_permit.so account required pam_permit.so password required pam_permit.so session required pam_permit.so
Then run init 3 to boot to multiuser mode. No sense trying to start KDE or Gnome because PAM is missing all those modules. Login from the console should be enabled now that we added the login module. Now the /etc/pam.d contents can be restored from backup or whatever else is needed. The temporary login module we show enables any valid user to access the system. This approach doesn't seem secure, but then again, the only place to log in is from the console. There is no ssh or other module. If other access points use login besides the console, though, this might not be the best choice. In that event, it might be better to stay in single user mode and restore the /etc/pam.d from there.
For whatever reason, it might not be possible to use the previous procedure in single user mode. Maybe Linux does not boot at all, or the root filesystem is read-only, and the /etc/pam.d/login file can't be created. If this happens, boot from a recovery CD or floppy. After the box is booted with this method, the repairs to /etc/pam.d can be made. Chapter 1 explains booting from recovery CDs and floppy disks.
The rpm command verifies that the files in a package are the same as when they were installed. If an application isn't acting right, and PAM is suspected, try verifying that the module is the same as when it was delivered. For example,
#rpm -V -f /etc/pam.d/login .......T c /etc/pam.d/login
The T means the timestamp has changed, and the c indicates /etc/pam.d/login is a configuration file. Please note that the previous syntax verifies the package that delivered the login file and not just the login file itself.
The rpm command is a valuable troubleshooting tool. If you have not read through the rpm(8) man page, you should consider it.
It can be tempting just to verify the PAM packages, but this approach does not prove that the modules are ok. The modules are delivered by the applications that use them. For example:
#rpm -q -f /etc/pam.d/login util-linux-2.11y-9.progeny.1 #rpm -q -f /etc/pam.d/sshd openssh-server-3.5p1-11.progeny.2 #rpm -q -f /etc/pam.d/samba samba-2.2.7a-8.9.0
Bugs in PAM
After ruling out installation and configuration issues, it might be time to look for known bugs. Search not only the PAM Web site but also the application sites. If you think you have found a problem with the sshd PAM module, check to see what package delivered it:
# rpm -q -f /etc/pam.d/sshd openssh-3.7.1p2-113
Look at the documentation supplied with the application to see whether this is a known issue:
# rpm -q --filesbypkg openssh-3.7.1p2-113 (some lines omitted) openssh /etc/pam.d/sshd (some lines omitted) openssh /usr/share/doc/packages/openssh openssh /usr/share/doc/packages/openssh/CREDITS openssh /usr/share/doc/packages/openssh/ChangeLog openssh /usr/share/doc/packages/openssh/LICENSE openssh /usr/share/doc/packages/openssh/OVERVIEW openssh /usr/share/doc/packages/openssh/README openssh /usr/share/doc/packages/openssh/README.SuSE openssh /usr/share/doc/packages/openssh/README.kerberos openssh /usr/share/doc/packages/openssh/RFC.nroff openssh /usr/share/doc/packages/openssh/TODO openssh /usr/share/man/man1/scp.1.gz openssh /usr/share/man/man1/sftp.1.gz openssh /usr/share/man/man1/slogin.1.gz openssh /usr/share/man/man1/ssh-add.1.gz openssh /usr/share/man/man1/ssh-agent.1.gz openssh /usr/share/man/man1/ssh-copy-id.1.gz openssh /usr/share/man/man1/ssh-keyconverter.1.gz openssh /usr/share/man/man1/ssh-keygen.1.gz openssh /usr/share/man/man1/ssh-keyscan.1.gz openssh /usr/share/man/man1/ssh.1.gz openssh /usr/share/man/man5/ssh_config.5.gz openssh /usr/share/man/man5/sshd_config.5.gz openssh /usr/share/man/man8/sftp-server.8.gz openssh /usr/share/man/man8/ssh-keysign.8.gz openssh /usr/share/man/man8/sshd.8.gz
Finally, search the Internet for known issues. For sshd, http://www.openssh.com/report.html and http://bugzilla.mindrot.org/ are good places to start.