Chapter 16: Securing RPC, ActiveX Controls, and DCOM

Chapter 16

Securing RPC, ActiveX Controls, and DCOM

Remote procedure calls (RPC) have been a backbone communications mechanism since the early days of Microsoft Windows NT 3.1 (way back in 1993). There are two main variants on RPC: DCE (Distributed Computing Environment) RPC and ONC (Open Network Computing) RPC. Both are open standards, and both are implemented on several platforms. DCE RPC is the version Microsoft platforms use, although many ONC RPC applications also run on Windows platforms. DCE RPC is sometimes called Microsoft RPC, and ONC RPC is sometimes called Sun RPC. In the remainder of this chapter, RPC will refer to Microsoft's implementation of DCE RPC, although some of the concepts apply to both types of RPC.

A large number of applications written for Windows NT and beyond rely heavily on the RPC infrastructure. Because security is all about strengthening every aspect of a system, it's imperative that your RPC applications be robust and secure from attack. This chapter also covers Distributed COM (DCOM) applications and ActiveX controls, primarily because RPC is a technology used by DCOM, which is the mechanism by which COM applications communicate, and ActiveX controls are a specific type of COM technology.

Keeping in mind that we should learn from past mistakes when designing and building secure applications, let's look at three RPC security vulnerabilities fixed by Microsoft. The first attack occurs when an attacker sends invalid data to the Local Security Authority (LSA), which causes the LSA to hang. On the surface, the bug looks like an API issue; however, the problem occurs because the LsaLookupSids API forwards malformed data to the LSA over RPC. You can read more about this in the Malformed Security Identifier Request vulnerability Microsoft Security Bulletin at http://www.microsoft.com/technet/security/bulletin/ms99-057.asp.

The second attack relates to sending garbage to port 135 on a computer running Windows NT 3.51 or Windows NT 4. Doing so causes the RPC server listening on this port to spin the CPU up to 100 percent, essentially denying access to the server to users. The most common way to perform this attack is to connect to port 135 by using a telnet client, type in more than 10 random characters, and disconnect. You can read more about this bug in the Microsoft Knowledge Base article titled Telnet to Port 135 Causes 100 Percent CPU Usage at http://support.microsoft.com/support/kb/articles/Q162/5/67.asp.

Finally, a Microsoft Security Bulletin released in July 2001, Malformed RPC Request Can Cause Service Failure, relates to RPC server stubs I'll explain these shortly not performing appropriate validation before passing requests to various services, thereby potentially enabling denial of service (DoS) attacks. You can read this bulletin at http://www.microsoft.com/technet/security/bulletin/ms01-041.asp.