Securing RPC, ActiveX Controls, and DCOM

Chapter 10

Securing RPC, ActiveX Controls, and DCOM

Remote procedure calls (RPC) have been a backbone communications mechanism since the early days of Microsoft Windows NT 3.1 (way back in 1993). A large number of applications written for Windows NT and beyond rely heavily on the remote procedure call (RPC) infrastructure. Because security is all about strengthening every aspect of a system, it is imperative that your RPC applications be robust and secure from attack. This chapter also covers Distributed COM (DCOM) applications and ActiveX controls, primarily because RPC is a technology used by DCOM, which is the mechanism by which COM applications communicate, and ActiveX controls are a specific type of COM technology.

Keeping in mind that we should learn from past mistakes when designing and building secure applications, let s look at three RPC security vulnerabilities fixed by Microsoft. The first attack occurs when an attacker sends invalid data to the Local Security Authority (LSA), which causes the LSA to hang. On the surface, the bug looks like an API issue; however, the problem occurs because the LsaLookupSids API forwards malformed data to the LSA over RPC. You can read more about this in the Malformed Security Identifier Request vulnerability Microsoft Security Bulletin at www.microsoft.com/technet/security/bulletin/ms99-057.asp.

The second attack relates to sending garbage to port 135 on a computer running Windows NT 3.51 or Windows NT 4. Doing so causes the RPC server listening on this port to spin the CPU up to 100 percent, essentially denying access to the server to users. The most common way to perform this attack is to connect to port 135 by using a telnet client, type in more than 10 random characters, and disconnect. You can read more about this bug in the Microsoft Knowledge Base article titled Telnet to Port 135 Causes 100 Percent CPU Usage at support.microsoft.com/support/kb/articles/Q162/5/67.asp.

Finally, a Microsoft Security Bulletin released in July 2001, Malformed RPC Request Can Cause Service Failure, relates to RPC server stubs I ll explain these shortly not performing appropriate validation before passing requests to various services, potentially enabling denial of service (DoS) attacks. You can read this bulletin at www.microsoft.com/technet/security/bulletin/ms01-041.asp.