Section 9.6. Using the Management Interface and Service Console to Administer the Virtual Infrastructure

9.6. Using the Management Interface and Service Console to Administer the Virtual Infrastructure

This section focuses on the day-to-day administration of your ESX server using the out-of-the-box administrative tools. Previous chapters of this book showed how to install ESX Server, create new VMs, and configure networking using the Management Interface. This chapter focuses on maintaining an existing ESX Server installation by modifying permissions on a virtual machine, integrating ESX Server with Active Directory, and monitoring performance. The specific tools we'll be using are the MUI and Service Console. Even if you have VirtualCenter deployed in your environment, some tasks can be accomplished just as easily with the MUI, Service Console, or Remote Console. You'll also find that certain tasks can be performed only from the MUI or Service Console. Examples include:

Setting advanced options like masking LUNs
Creating and formatting new VMFS volumes
Creating and activating SWAP files
Viewing system logs files
Modifying the system startup profile

9.6.1. Virtual Machine Security in ESX Server

The ESX Server security model is much different from the VirtualCenter Security model. For one, security is not inherited, so it must be explicitly defined. Also, the permissions are applied differently. On ESX Server a directory/file has an owner (User), an affiliated group, and Others (everyone else). An example of the permissions is shown in Figure 9.142 In this example the user who owns the file is root, and the affiliated group is root. The user has read, write, and execute permissions, and the group and others have read and execute permisssions.

Figure 9-142. ESX Server Permissions

Table 9.9 provides a breakdown of each set of permissions and what they will allow you to perform on a virtual machine.

Table 9-9. ESX Server permissions
Permissions on VM files	Rights
R	View the state of VM in the MUI
W	Modify VM hardware
X	Perform power operations on a VM and use the Remote Console

For users or groups to fully administer a virtual machine, they will need to have read, write, and execute permissions on the configuration files and read and write permissions on the virtual disk (the execute permissions has no effect on .vmdk files, but you could set the permissions the same just for consistency). Table 9.10 summarizes what file permissions are required to allow or restrict what actions a user or group can perform on a virtual machine. To achieve similar functionality of VirtualCenter with ESX Server, you can implement permissions in the following way.

Table 9-10. ESX Default Roles for Controlling Access to VMs
User Role	Permissions	Privileges
Root Usersimilar to VirtualCenter	Full permissions to all files and directories	Can perform all necessary actions on ESX Server and Administrator VMs.
Owner(User) similar to Virtual Machine Administrator	R+W+X on configuration files (VMX) and R+W on virtual disk (VMDK) files.	Can create new VMs; make changes to VM hardware; and stop, start, reset, and suspend VMs through MUI, Remote Console, or API. Access the VM with Remote Console. View the VM in the MUI.
Group(affiliated group)similar to Virtual Machine User	R+X on configuration files and R+X on virtual disk (VMDK) files.	Start, Stop, Reset and Suspend VM through the MUI, Remote Console, or API. Access the VM with Remote Console and view the VM in the MUI.
Other (everyone else)similar to Read-Only User	R on configuration files and R on virtual disk (VMDK) files	View the VM in the MUI.

9.6.1.1. Viewing and Editing File Permissions with the MUI

If you need to change permissions on a file or directory, you can use the MUI or the Service Console. Refer to Chapter 13 for the command-line options for changing permissions in the Service Console. To change permissions in the MUI, click on the Manage Files hyperlink shown in Figure 9.143

Figure 9-143. Manage Files from the MUI

You can then use the MUI Web interface to browse to the file or directory you need to modify permissions on. You'll need to select the checkbox next to the file or directory then click Edit Properties at the bottom of the screen shown in Figure 9.144

Figure 9-144. The Manage Files MUI Interface

After you've clicked Edit Properties, you'll see a screen like the one shown in Figure 9.145 From this screen you can change the user (owner) or group affiliated with the file and modify the permissions for User, Group or Others.

Figure 9-145. The File Properties Screen

9.6.2. Creating New Users and Groups or Modifying Existing Users and Groups on ESX Server

If you need to add new users or groups to your ESX Server, you can use the MUI or the Service Console. Refer to Chapter 13 for the command-line options for adding users and groups through the Service Console. To add users or groups with the MUI, start by clicking on the Option tab like the one shown in Figure 9.146 Click on the Users and Groups… hyperlink to start the Users and Group applet shown in Figure 9.147

Figure 9-146. MUI Options Tab

9.6.2.1. Modifying Existing Users or Creating New Users

To modify an existing user, click on the username shown in Figure 9.147 You'll be able to modify the home directory, change the password, and modify group membership or remove (delete) the user.

To add a new user, click on the Add hyperlink, as shown in Figure 9.147

Figure 9-147. Users and Groups

In Figure 9.148 you can create a new user and add the user to existing groups on the server. Click OK when finished.

Figure 9-148. Add a New User to ESX Server

9.6.2.2. Modifying Existing Groups or Creating New Groups

To modify an existing group, click on the group name in the Users and Groups applet. You'll have the option to add or remove members or remove (delete) the group. To add a new group, click the Add hyperlink under Groups in the Users and Groups applet shown in Figure 9.149

Figure 9-149. Add New Groups

After you click the Add button, you'll see a screen like the one shown in Figure 9.150 You must specify a group name and add members to the group. When you are done, click OK.

Figure 9-150. Create a New Group and Add Members

9.6.3. Integrating ESX Server with Active Directory

Because a large majority of IT shops that will deploy ESX server already have Active Directory in their environments, it would be a good idea to simplify administration by allowing virtual machine administrators to log in with existing Active Directory credentials. This will be more valuable if you won't have VirtualCenter deployed in your environment. This section will show you the steps required to configure the Pluggable Authentication Module (PAM) to work with Active Directory.

9.6.3.1. Configuring Your PAM to Work with Active Directory

We will be creating and or updating the following files to achieve Active Directory authentication within ESX Server:

/etc/krb5.conf (krb5.conf file need to be created)
/var/kerberos/krb5kdc/kdc.conf (Kerberos and krb5kdc directories need to be created along with kdc.conf file)
/etc/pam.d/vmware-authd (vmware-authd will need to be modified)

The first file you need to create and edit is /etc/krb5.conf. Please refer to Chapter 13 if you need assistance with creating text files in the Service Console. It covers both vi and nano. Figure 9.151 shows the contents of /etc/krb5.conf, and we have highlighted the text that will need to change for your environment. In the example in Figure 9.151, every reference to virtual.net should be changed to reflect your Active Directory domain name, and any reference to server-01 should be changed to the name of one of your Active Directory domain controllers.

Figure 9-151. Contents of the /etc/krb5.conf File

You need to create the file shown in Figure 9.152 and make sure you replace the domain name that is highlighted with your Active Directory Domain name.

Figure 9-152. Contents of the /var/kerberos/krb5kdc/kdc.conf File

You also need to edit the /etc/pam.d/vmware-authd file. Make sure your file looks like the one shown in Figure 9.153 We have highlighted the text that will need to be changed or added to the file.

Figure 9-153. Contents of the /etc/pam.d/vmware-authd File

Configuring & Implementing…
Creating User Accounts on Your ESX Server

Once you have configured the PAM for Active Directory authentication, you'll need to create local user accounts that map to Active Directory accounts using the Service Console adduser command. If you use the MUI to create a user account, it will force you to specify a password that will create and store a local password. When the users logs in to ESX Server, it will try to authenticate them locally rather than using their Active Directory accounts and passwords. When using the adduser command, you don't have to create a password for the account (This is not the same as a blank password). For more information on the adduser command, refer to Chapter 13 or type man adduser at the command line of the Service Console.

Changing the Time of Your Service Console

If you have not configured you Service Console to sync time with an NTP time source (this topic is covered in Chapter 13), you may find it necessary to change the system clock on your server to match you Active Directory Domain controllers to make sure it falls within the tolerance levels of Kerberos authentication.

9.6.4. Performance Monitoring and Management through ESX Server

This section covers the different options you have available for monitoring ESX Server and virtual machines. VirtualCenter enables you to perform actions when alarms are triggered. You don't get those options using out-of-the-box tools, but you still have the ability to monitor performance and make adjustments to properly tune your virtual machines. You also have third-party system management tools from the major hardware vendors that can assist with proactively monitoring your ESX Server. The following tools can be used to monitor the performance of your ESX Server and VMs:

The MUI Status Monitor and Memory Pages
The esxtop utility
The vmkusage utility

9.6.4.1. Performance Monitoring

VMware has several interfaces that can be used to monitor the overall performance of you ESX Server and virtual machines. The main areas of concern for performance are CPU, memory, disk I/O, and network I/O. When measuring CPU performance, you have the physical CPU, the logical CPU (for servers with hyperthreading), and the virtual CPU. When monitoring memory, you have physical memory and memory allocated to VMs. Disk I/O and network I/O can both be measured from the physical interfaces and the traffic generated by the VMs themselves.

9.6.4.1.1. MUI Status Monitor

The status monitor provides a high-level overview of resource utilization on your ESX Server and individual VMs (see Figure 9.154). These utilization statistics are averaged over five-minute periods and don't provide historical data for trending. In the system summary section of the status monitor, you can get an overall view of CPU utilization and memory utilization broken down by virtual machine and system services. In the virtual machines section of the status monitor, you can view the uptime of each VM, the total number of registered VMs on the server, their current CPU utilization, and the amount of configured RAM.

Figure 9-154. The ESX Server MUI Status Monitor

Figure 9.155 shows detailed memory utilization statistics for both the service console and virtual machines. The statistics are averaged over five-minute periods. The screen in Figure 9.155 can be used to view the availability of memory of your server. If you start seeing excessive swap file usage, you either need to move some of the VMs to another host or install more memory in your server. The graphical bars will show you how much memory is being utilized for the system as a whole and for each individual VM. Transparent page sharing is a cool feature of ESX Server that allows VMs to share memory pages that have identical content. If a VM needs to modify the shared data, then the VMkernel creates a new copy for the VM that it can modify. The operating systems are not aware that they are sharing memory with other VMs (hence, the name - transparent page sharing). The reason we point this out is because one of the utilization statistics shown in Figure 9.155 called Shared is the amount of shared memory. The shared memory is referring to the amount of transparent page sharing that is taking place on the server. This is a good feature.

Figure 9-155. The ESX Server MUI Memory Page

9.6.4.1.2. esxtop

The esxtop utility provides real-time performance statistics that can be reviewed to determine if bottlenecks exist on your server. Figure 9.156 shows the output of the esxtop utility. Some of the main values to look for are %Ready values for VMs. They should remain below 5 percent under normal circumstances. %Ready is the amount of time a VM was ready for processing, but could not get scheduled to run on the physical processor. If you have VMs that have a %Ready value greater than 5%, you may need to migrate the VM to another host or adjust the minimum amount of CPU time or shares for that VM. The line labeled PCPU in Figure 9.156 displays the utilization for each physical CPU in the system and the aggregate total for both. The line below that labeled LCPU shows the utilization for each logical CPU on the server (if hyperthreading is enabled). As you may expect, if you were to add the utilization percentage for each logical CPU, you would still get the aggregate total for both of the physical CPUs in the server. The next two lines, MEM and SWAP, relate to system memory. In the example from Figure 9.156, the physical memory is 70% utilized, and there is currently no SWAP being used. If the SWAP is being used heavily for overcommitment of memory, you need to consider adding more memory or moving some VMs to another host. If you need to find out specific information about a VM with esxtop, you'll need to get the VMID (World ID) for the VM. This can be found in the MUI under the display name of the virtual machine. Once you have located the VMID, you can find the specific performance statistics for the VM in esxtop. For example, one of the machines listed back in Figure 9.154 has a VMID of 143, and it also has two VCPUs. If you look you'll find two entries under the WID column for 143 in Figure 9.156 You can now view performance statistics for each virtual CPU of a specific VM.

Figure 9-156. The Output of the esxtop Utility

9.6.4.1.3. vmkusage

With VMware ESX Server 2.5, vmkusage is available and ready to install. To use vmkusage for the first time, you need to log in to the Service Console and run the following command:

 vmkusagectl install

Afterward, log in to the MUI and specify the URL to your ESX Server followed by vmkusage; for example, https://esxserver.domain.net/vmkusage.

Figure 9.157 shows the home page of the vmkusage Web application. You can view recent, daily, and weekly performance data for the following components:

Storage disk statistics
Network utilization
CPU utilization
Memory utilization

The same concepts apply with this tool as with all the other tools. You need to look at CPU utilization and available memory. Is your ESX Server using the SWAP file because there is not enough physical memory for all your VMs? You can explore this tool and obtain a ton of performance data for the system as a whole and each individual VM. You need to know the maximum throughput of your HBA/disk subsystem so that you can gauge whether you disk I/O is becoming a bottleneck for your system.

Figure 9-157. The Main Screen for the vmkusage Web Application

9.6.4.2. Performance Tuning

ESX Server has several mechanisms that can be used to ensure that VMs are allocated adequate resources. ESX Server also allows you to guarantee the amount of CPU, memory, disk I/O, and network bandwidth allocated to a VM. You also have the ability to specify which physical or logical processor a VM will run on. Figure 9.158 shows the CPU Resource Settings applet for a VM. If you look under scheduling affinity, you have the option to control which processors a VM runs on. By default, the VMkernel dynamically schedules virtual machines and the Service Console on physical CPUs every 20 milliseconds and looks to migrate VMs to other physical CPUs when required. (The Service Console always runs on CPU 0 and does not migrate).

Figure 9-158. CPU Resource Settings

9.6.4.2.1. How Proportional Shares Work

The way shares work in an ESX Server is similar to shares of stock issued by a company. The more shares a person owns, the more power he or she has to influence the company. To translate that to VMware terms, the more shares a VM has for a resource, the more access it will have to that resource during times of contention. If you have 10 VMs on your server, and eight of them have 2,000 shares allocated for CPU, and two of them have 4,000 shares, you have the following total shares available:

(8*2,000)+(2*4,000) = 24,000 outstanding CPU shares
Each VM with 2,000 shares has 8.3 percent of the available shares (2,000/24,000 *100 = 8.3)
Each VM with 4,000 shares has 16.6 percent of the available shares (4,000/24,000 *100= 16.6)

So the VMs with 4,000 shares are guaranteed 16.6% of CPU time when there is contention for CPU time, and the VMs with 2,000 shares will be guaranteed 8.3% of CPU time during times of contention. This same concept translates to memory and disk shares as well.

If you look back at Figure 9.140, you'll notice minimum and maximum settings for CPU. This is where you can guarantee that a VM will receive no less or no more than the minimum or maximum settings that you specify for a resource. If you try to power on a VM, and the minimum requirements you specified cannot be met, you won't be able to power it on.