The Compaq/HP document dedicated to system performance is called Open VMS Performance Management. This book describes the tuning algorithm extensively. It also describes how to use the performance programs available with OpenVMS.
Monitoring and tuning TCP/IP are described in Compaq TCP/IP Services for OpenVMS Tuning and Troubleshooting.
The Compaq Availability Manager User's Guide is at. It can be run from either an OpenVMS terminal or a Windows terminal.
DECevent is described in the OpenVMS System Management Utilities Reference Manual.
The OpenVMS System Manager's Manual, Chapter 15, discusses AUTOGEN usage in detail. The OpenVMS System Management Utilities Reference Manual details its options.
My book, OpenVMS Operating System Concepts, describes paging and thrashing in great detail.
Coburn, James W. OpenVMS Performance Management, second edition. This book addresses specific algorithms used to test and adjust OpenVMS. Because it is dated, this text should be used in conjunction with OpenVMS Performance Management.